Data¶
The historical data for this project is available in file https://archive.ics.uci.edu/ml/datasets/Bank+Marketing
Attributes related to Bank Client Data¶
- age: Continuous feature
- job: Type of job (management, technician, entrepreneur, blue-collar, etc.)
- marital: marital status (married, single, divorced)
- education: education level (primary, secondary, tertiary)
- default: has credit in default?
- housing: has housing loan?
- loan: has personal loan?
- balance in account
Attributes related to previous contact¶
- contact: contact communication type
- month: last contact month of year
- day: last contact day of the month
- duration: last contact duration, in seconds*
Other Attributes¶
- campaign: number of contacts performed during this campaign and for this client
- pdays: number of days that passed by after the client was last contacted from a previous campaign (-1 tells us the person has not been contacted or contact period is beyond 900 days)
- previous: number of contacts performed before this campaign and for this client
- poutcome: outcome of the previous marketing campaign
Output Variable / Target:¶
- Target: Tell us has the client subscribed a term deposit. (Yes, No)
In [1]:
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import pandas_profiling
import pydotplus as pydot
import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline
sns.set(color_codes=True)
from sklearn import tree
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn import metrics
from sklearn.metrics import accuracy_score,f1_score,recall_score,precision_score, confusion_matrix # for model validation scores
from sklearn.feature_extraction.text import CountVectorizer # DT does not take strings as input for the model fit step
from yellowbrick.classifier import ClassificationReport, ROCAUC
import missingno as msno_plot # for plotting missing values
from os import system
from IPython.display import display # for displaying multiple data frames in one output
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))
plt.style.use('ggplot')
pd.options.display.float_format = '{:,.4f}'.format
1 Basic EDA & Univariate Analysis¶
- Read the data (csv file)
- Shape
- Column Names (Attributes)
- Info (Counts / Types)
- Head & Tail (first 5 / last 5 rows of data)
High Level View of Dataset¶
In [2]:
df = pd.read_csv("bank-full.csv")
print(df.columns)
print()
print(df.shape)
print()
print(df.info())
print()
print(df.size)
print()
pd.concat([df.head(5),df.tail(5)])
Out[2]:
Reorder Attributes to Place Object Types on "Right of Dataframe"¶
In [3]:
newOrder=['age','balance','day','duration','campaign','pdays','previous','job','marital','education','default','housing','loan','contact','month','poutcome','Target']
df=df[newOrder]
pd.concat([df.head(5),df.tail(5)])
Out[3]:
Overview of Numerical and Object-Type Attributes¶
In [4]:
print("General information:")
df.info()
print()
print("Unique values per columns:")
df.nunique()
Out[4]:
In [5]:
df.describe().T
Out[5]:
In [6]:
df.describe(include='object')
Out[6]:
General Findings & Comments:
- Records:45,211
- Attributes: 17
- Seven (7) Attributes of Type Int64
- Ten (10) Attributes of Type Object that may need to be converted to Ordinal Type
- Attributes 'pdays' and 'previous' have equal Q1 and Q3
Actions:
- Replace pdays' values of -1 with values of 0
- Check for and Treat Missing Values
- Check for and Treat outliers in Numerical Attributes
Replace values in attributes of interest¶
In [7]:
# Replacing pdays value of -1 with value of 0
df['pdays']=df['pdays'].replace(-1, 0)
df.describe().T
Out[7]:
Check for Missing Values & Treat if necessary¶
Missing Value Check¶
In [8]:
print("Percent of missing data:")
print(df.isnull().mean()*100)
print()
print("Visual representation of missing data:")
msno_plot.bar(df);
Missing Value Treatement¶
None Necessary
Check for Outliers & Treat if Necessary¶
Check for Outliers in Numerical Attributes¶
In [9]:
# check for outliers in attributes with continuous numerical values
col_num= ['age', 'balance', 'day', 'duration', 'campaign', 'pdays', 'previous']
plt.figure(figsize=(20,15))
pos = 1
for i in col_num:
plt.subplot(6, 7, pos)
sns.boxplot(df[i])
pos += 1
print()
for i in col_num:
plt.subplot(5, 7, pos)
sns.distplot(df[i])
pos += 1
In [10]:
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
In [11]:
# focused analysis on attributes with Q1 = Q3 vs Target
sns.scatterplot(df['previous'], df['pdays'], hue=df['Target'], palette='Set2')
Out[11]:
In [12]:
sns.distplot(df[df['Target']=="no"]['previous'],color='r',label="no")
sns.distplot(df[df['Target']=="yes"]['previous'],color='g',label="yes")
plt.legend()
plt.show()
In [13]:
sns.distplot(df[df['Target']=="no"]['pdays'],color='r',label="no")
sns.distplot(df[df['Target']=="yes"]['pdays'],color='g',label='yes')
plt.legend()
plt.show()
In [14]:
df.groupby(['Target','pdays'])['previous'].value_counts()
Out[14]:
In [15]:
df.groupby(['Target','previous'])['pdays'].value_counts()
Out[15]:
Outlier Treatement for 'pdays' and 'previous' - remove columns entirely as > 90% of values where these are 0, "Target" = "yes", meaning that for more than 90% of time, a client subscribed without any prior / recent contact.
For all other attributes, remove outliers per 1.5XQ1/Q3 rules.
Remove Outliers from Numerical Attributes¶
In [16]:
# image included for context
# image acquired from blog post "Understanding Boxplots" by Michael Galarnyk at
# https://towardsdatascience.com/understanding-boxplots-5e2df7bcbd51
from IPython.display import Image
Image("boxplotOutliersDist.jpg")
Out[16]:
In [17]:
# remove outliers
for i in col_num:
q1, q2, q3 = df[i].quantile([0.25,0.5,0.75])
IQR = q3 - q1
lower_cap=q1-1.5*IQR
upper_cap=q3+1.5*IQR
df[i]=df[i].apply(lambda x: upper_cap if x>(upper_cap) else (lower_cap if x<(lower_cap) else x))
# confirm outliers were removed
plt.figure(figsize=(20,15))
pos = 1
for i in col_num:
plt.subplot(6, 7, pos)
sns.boxplot(df[i])
pos += 1
print()
for i in col_num:
plt.subplot(5, 7, pos)
sns.distplot(df[i])
pos += 1
In [18]:
# drop 'pdays' and 'prevoius' columns
col_drop= ['pdays', 'previous']
df=df.drop(labels=col_drop, axis='columns')
df.describe().T
#df = df.drop(['pdays','previous'],axis=1)
Out[18]:
General Findings & Comments:
- No Missing Values --> No Treatment
- Attributes 'pdays' and 'previous' have equal Q1 and Q3 and >90% of "Target"="yes" records derived from no previous or recent contacts made, so contact made as irrelevant to producing a "Target" of "yes" (client subscription). These columns were dropped.
- All other numerical attributes were treated for outliers.
Actions:
- Attribute Conversion - object to categorical
- Categorical Attribute Encoding - ordinal -> numerical, yes/no --> binary
Attributes that may require Binning (over 12 unique values)
- age
- balance
- day
- duration
- campaign
Attributes that may require conversion binary values of 0 or 1 for yes/no values
- default
- housing
- loan
- Target
Attributes that may require ordinal/numerical encodings
- month
- poutcome
Attributes that may require one-hot encodings
- job
- contact
- marital
- education
Attribute Type Conversion - Object to Categorical¶
In [19]:
for feature in df.columns: # Loop through all columns in the dataframe
if df[feature].dtype == 'object': # Only apply for columns with categorical strings
df[feature] = pd.Categorical(df[feature])# Replace strings with an integer
print(df.info())
pd.concat([df.head(5),df.tail(5)])
Out[19]:
Conversions of Categories (text/string) to Numerical Values:¶
- Create "binary attributes": Convert attributes with "yes" or "no" to binary (0 or 1)
- Create "ordinal attributes": Convert attributes with text "values" of ordered categories to numerical
- Create "one-hot-encoded attributes: Convert attributes with text "values" of no clear/direct order to one-hot endcoded variables
By Attribute:
- default (binary)
- housing (binary)
- loan (binary)
- Target (binary)
- month (ordinal encoding)
- poutcome (ordinal encoding)
- job (one-hot-encoding)
- contact (one-hot-encoding)
- marital (one-hot-encoding)
- education (one-hot-encoding)
In [20]:
# counts of distinct vaues in categorical attributes
print('For job:')
print(df.job.value_counts())
print()
print('For marital:')
print(df.marital.value_counts())
print()
print('For education:')
print(df.education.value_counts())
print()
print('For default:')
print(df.default.value_counts())
print()
print('For housing:')
print(df.housing.value_counts())
print()
print('For loan:')
print(df.loan.value_counts())
print()
print('For contact:')
print(df.contact.value_counts())
print()
print('For month:')
print(df.month.value_counts())
print()
print('For poutcome:')
print(df.poutcome.value_counts())
print()
print('For Target:')
print(df.Target.value_counts())
print()
# need to: create iterative function
In [21]:
df.pivot_table(index='job', columns='Target', values='age').astype('int')
#turning numbers into integers
Out[21]:
In [22]:
df.pivot_table(index='job', columns='Target', values='age', aggfunc='count')
Out[22]:
In [23]:
df.pivot_table(index='marital', columns='Target', values='age').astype('int')
Out[23]:
In [24]:
df.pivot_table(index='marital', columns='Target', values='age', aggfunc='count')
Out[24]:
In [25]:
df.pivot_table(index='education', columns='Target', values='age').astype('int')
Out[25]:
In [26]:
df.pivot_table(index='education', columns='Target', values='age', aggfunc='count')
Out[26]:
In [27]:
replaceStruct = {
"month": {"jan":1,"feb":2,"mar":3,"apr":4,"may":5,"jun":6,"jul":7,"aug":8,"sep":9,"oct":10,"nov":11,"dec":12},
"poutcome": {"failure": 0, "unknown": 1, "other": 2, "success": 3},
"default": {"no": 1, "yes": 0 },
"housing": {"no": 0, "yes": 1 },
"loan": {"no": 0, "yes": 1 },
"education": {"unknown": 0, "primary": 1, "secondary": 2, "tertiary": 3 },
#"marital": {"divorced": 0, "single": 1, "married": 2 },
"contact": {"unknown": 0, "telephone": 1, "cellular": 2 },
#"job": {}
"Target": {"no": 0, "yes": 1 }
}
ohe_cols=["job","marital"]
df=df.replace(replaceStruct)
df=pd.get_dummies(df, columns=ohe_cols)
df.info()
pd.concat([df.head(5),df.tail(5)])
Out[27]:
In [28]:
dfnum=df[['age', 'balance', 'day', 'duration', 'campaign']]
dfnum.describe().T
Out[28]:
In [29]:
dfnum.nunique()
Out[29]:
In [30]:
# distributions of numerical (continuous) attributes
fig, axs = plt.subplots(ncols = 5, figsize = (20, 5))
bins=5
sns.distplot(df['age'], bins=bins, ax = axs[0])
sns.distplot(df['balance'], bins=bins, ax = axs[1])
sns.distplot(df['day'], bins = bins, ax = axs[2])
sns.distplot(df['duration'], bins = bins, ax = axs[3])
sns.distplot(df['campaign'], bins=bins, ax = axs[4]);
#sns.distplot(df['pdays'], bins=30, ax = axs[5]);
#sns.distplot(df['previous'], bins=30,ax = axs[6]);
# Boxplots
col_num= ['age', 'balance', 'day', 'duration', 'campaign']
plt.figure(figsize=(29,5))
pos = 1
for i in col_num:
plt.subplot(5, 7, pos)
sns.boxplot(df[i])
pos += 1
In [31]:
df.describe().T
Out[31]:
In [32]:
df.columns
Out[32]:
In [33]:
# distributions of numerical (continuous) attributes
fig, axs = plt.subplots(ncols = 5, figsize = (30, 7))
bins=5
sns.distplot(df['age'], bins=6, ax = axs[0])
sns.distplot(df['balance'], bins= bins,ax = axs[1])
sns.distplot(df['day'], bins = 4, ax = axs[2])
sns.distplot(df['duration'], bins = 11, ax = axs[3])
sns.distplot(df['campaign'], bins = 2,ax = axs[4]);
#sns.distplot(df['pdays'], bins=bins, ax = axs[5]);
#sns.distplot(df['previous'], bins=bins,ax = axs[6]);
In [34]:
# distributions of ordinal attributes
fig, axs = plt.subplots(ncols = 5, figsize = (30, 7))
bins=5
sns.distplot(df['month'], bins = bins, ax = axs[3])
sns.distplot(df['default'], bins=2, ax = axs[0])
sns.distplot(df['housing'], bins = 2, ax = axs[1])
sns.distplot(df['loan'], bins = 2,ax = axs[2])
sns.distplot(df['poutcome'], bins = 4, ax = axs[4]);
2 Bivariate & Multivariate Analysis:¶
In [35]:
df.info()
In [36]:
df.describe().T
Out[36]:
Correlation plot - Understanding degree of association among different variables¶
In [37]:
# correlations of Numerical / Continuous attributes and Target
df1=df[['age', 'balance', 'day', 'duration', 'campaign','Target']]
sns.heatmap(df1.corr(),
annot=True,
linewidths=.5,
center=0,
cbar=False,
cmap="YlGnBu")
plt.show()
df1.corr()
Out[37]:
In [38]:
# correlations of One Hot Encoded 'job' attributes and Target part 1
df2=df[['job_admin.', 'job_blue-collar','job_entrepreneur', 'job_housemaid', 'job_management', 'job_retired','Target']]
plt.figure(figsize=(10,8))
sns.heatmap(df2.corr(),
annot=True,
linewidths=.5,
center=0,
cbar=False,
cmap="YlGnBu")
plt.show()
df2.corr()
Out[38]:
In [39]:
# correlations of One Hot Encoded 'job' attributes and Target part 2
df3=df[['job_self-employed', 'job_services', 'job_student', 'job_technician','job_unemployed', 'job_unknown','Target']]
plt.figure(figsize=(10,8))
sns.heatmap(df3.corr(),
annot=True,
linewidths=.5,
center=0,
cbar=False,
cmap="YlGnBu")
plt.show()
df3.corr()
Out[39]:
In [40]:
# correlations of One Hot Encoded 'marital' attributes and Target
df4=df[[ 'marital_divorced',
'marital_married', 'marital_single','Target']]
plt.figure(figsize=(10,8))
sns.heatmap(df4.corr(),
annot=True,
linewidths=.5,
center=0,
cbar=False,
cmap="YlGnBu")
plt.show()
df4.corr()
Out[40]:
In [41]:
sns.pairplot(df1, hue="Target", palette="husl")
Out[41]:
In [42]:
sns.pairplot(df2, hue="Target", palette="husl")
Out[42]:
In [43]:
sns.pairplot(df3, hue="Target", palette="husl")
Out[43]:
In [44]:
sns.pairplot(df4, hue="Target", palette="husl")
Out[44]:
General Findings & Comments
- Is there correlation between variables whose values are >0.8? No
- Is there linear relationships between any two variables? No
- Is there a strong correlation between any two variables? Only between Martial Single and Marital Married (inverse relationship as expected / ignore)
- Is there a strong correlation between any independent variable and Target variable? NO
- "job" and "marital" related attributes have negligible impact on "Target" = "yes" --> drop all "job" and 'marital' related attributes
3 Data Preparation:¶
xxx
In [45]:
df.columns
Out[45]:
In [46]:
df = df.drop([ 'job_admin.', 'job_blue-collar', 'job_entrepreneur', 'job_housemaid',
'job_management', 'job_retired', 'job_self-employed', 'job_services',
'job_student', 'job_technician', 'job_unemployed', 'job_unknown',
'marital_divorced', 'marital_married', 'marital_single'],axis=1)
In [47]:
df_logr=df # dataframe for logistic regression model
df_tree=df # dataframe for decision tree model
df_forest=df # dataframe for random forest model
df_bag=df # dataframe for bagging model
df_ada=df # dataframe for ada boost model
df_gradb=df # dataframe for gradient boost model
4 Logistic Regression Model:¶
In [48]:
X = df_logr.drop(['Target'], axis=1)
Y = df_logr[['Target']]
##Split into training and test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.30,random_state=42)
In [49]:
print("{0:0.2f}% data is in training set".format((len(X_train)/len(df_logr.index)) * 100))
print("{0:0.2f}% data is in test set".format((len(X_test)/len(df_logr.index)) * 100))
In [50]:
from sklearn.metrics import confusion_matrix, recall_score, precision_score, f1_score, roc_auc_score, accuracy_score, log_loss
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(random_state=42);
logreg.fit(X_train, y_train);
In [51]:
logreg
Out[51]:
In [52]:
logreg.coef_
Out[52]:
In [53]:
logreg.intercept_
Out[53]:
In [54]:
y_predict = logreg.predict(X_test)
y_predict = logreg.predict(X_train) #Prediction here are 0s and 1s
y_predict_prob = logreg.predict_proba(X_train) #to get probability values
In [55]:
y_predict = logreg.predict(X_train) # Prediction here are 0s and 1s
y_predict_prob = logreg.predict_proba(X_train)
In [56]:
## function to get confusion matrix in a proper format
def draw_cm( actual, predicted):
cm = confusion_matrix(actual, predicted)
print(cm)
sns.heatmap(cm, annot=True, fmt='.2f', xticklabels = [0,1] , yticklabels = [0,1] )
plt.ylabel('Observed')
plt.xlabel('Predicted')
plt.show()
In [57]:
confusion_matrix(y_train,y_predict)
Out[57]:
In [58]:
# Regression Score
print("Trainig accuracy",logreg.score(X_train,y_train))
print()
print("Testing accuracy",logreg.score(X_test, y_test))
print()
print('Confusion Matrix')
print(draw_cm(y_train,y_predict))
print()
print("Recall:",recall_score(y_train,y_predict))
print()
print("Precision:",precision_score(y_train,y_predict))
print()
print("F1 Score:",f1_score(y_train,y_predict))
print()
In [59]:
print("Roc Auc Score:",roc_auc_score(y_train,y_predict))
In [60]:
#AUC ROC curve
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
logit_roc_auc = roc_auc_score(y_train, logreg.predict(X_train))
fpr, tpr, thresholds = roc_curve(y_train, logreg.predict_proba(X_train)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('Log_ROC')
plt.show()
In [61]:
# print thresholds
fpr, tpr, thresholds =roc_curve(y_train, logreg.predict_proba(X_train)[:,1])
FPR=pd.DataFrame(fpr)
FPR.rename(columns = {0:'False Positive Rate'}, inplace = True)
TRP=pd.DataFrame(tpr)
TRP.rename(columns = {0:'True Positive Rate'}, inplace = True)
THRESHOLD=pd.DataFrame(thresholds)
THRESHOLD.rename(columns = {0:'Threshold'}, inplace = True)
result = pd.concat([FPR, TRP, THRESHOLD], axis=1, sort=False)
result.head(10)
Out[61]:
In [62]:
# Goal/Objective = threshold resulting in high TPR is high and FPR is low / where tpr - (1-fpr) is zero or near to zero is the optimal cut off point
import pylab as pl
i = np.arange(len(tpr)) # index for df
roc = pd.DataFrame({'fpr' : pd.Series(fpr, index=i),'tpr' : pd.Series(tpr, index = i), '1-fpr' : pd.Series(1-fpr, index = i), 'tf' : pd.Series(tpr - (1-fpr), index = i), 'thresholds' : pd.Series(thresholds, index = i)})
roc.iloc[(roc.tf-0).abs().argsort()[:1]]
# Plot tpr vs 1-fpr
fig, ax = pl.subplots()
pl.plot(roc['tpr'])
pl.plot(roc['1-fpr'], color = 'red')
pl.xlabel('1-False Positive Rate')
pl.ylabel('True Positive Rate')
pl.title('Receiver operating characteristic')
ax.set_xticklabels([])
display(roc.head())
In [63]:
roc[(roc['tpr']>0.6) & (roc['tpr']<0.7)].head(50)
Out[63]:
In [64]:
clf = LogisticRegression(random_state=42)
clf.fit(X_train, y_train)
THRESHOLD = 0.734359
preds = np.where(clf.predict_proba(X_test)[:,1] > THRESHOLD, 1, 0)
pd.DataFrame(data=[accuracy_score(y_test, preds), recall_score(y_test, preds),
precision_score(y_test, preds), roc_auc_score(y_test, preds)],
index=["accuracy", "recall", "precision", "roc_auc_score"])
Out[64]:
In [65]:
logreg.get_params() # check parameters of logistic regression. If we dont specify the parameters in the model it takes default value
Out[65]:
In [66]:
# Run loop to check different values of 'solver'. Note that solver can only be used with l2, only 'liblinear' works with both 'l1' and 'l2','liblinear','sag','saga'
train_score=[]
test_score=[]
solver = ['newton-cg','lbfgs', 'liblinear','sag','saga']
for i in solver:
model = LogisticRegression(random_state=42, penalty='l2', C = 0.75,solver=i) # changing values of solver
model.fit(X_train, y_train)
y_predict = model.predict(X_test)
train_score.append(round(model.score(X_train, y_train),3))
test_score.append(round(model.score(X_test, y_test),3))
print(solver)
print()
print(train_score)
print()
print(test_score)
In [67]:
model = LogisticRegression(random_state=42,penalty='l2') # change l1 penalty
model.fit(X_train, y_train)
y_predict = model.predict(X_test)
print("Trainig accuracy",model.score(X_train,y_train))
print()
print("Testing accuracy",model.score(X_test, y_test))
In [68]:
model = LogisticRegression(random_state=42, penalty='l2',class_weight='balanced') # changing class weight to balanced
model.fit(X_train, y_train)
y_predict = model.predict(X_test)
print("Trainig accuracy",model.score(X_train,y_train))
print()
print("Testing accuracy",model.score(X_test, y_test))
In [69]:
# Run loop to check different values of 'C'
train_score=[]
test_score=[]
C = [0.01,0.1,0.25,0.5,0.75,1]
for i in C:
model = LogisticRegression(random_state=42,penalty='l2', class_weight='balanced', C=i) # changing values of C
model.fit(X_train, y_train)
y_predict = model.predict(X_test)
train_score.append(round(model.score(X_train,y_train),3)) # appending training accuracy in a blank list for every run of the loop
test_score.append(round(model.score(X_test, y_test),3)) # appending testing accuracy in a blank list for every run of the loop
print(C)
print()
print(train_score)
print()
print(test_score)
In [70]:
# Final model
model = LogisticRegression(random_state=42,penalty='l2', class_weight='balanced',C=0.25)
model.fit(X_train, y_train)
y_predict = model.predict(X_test)
print("Trainig accuracy",model.score(X_train,y_train))
print()
print("Testing accuracy",model.score(X_test, y_test))
print()
print('Confusion Matrix')
print(draw_cm(y_test,y_predict))
print()
print("Recall:",recall_score(y_test,y_predict))
print()
print("Precision:",precision_score(y_test,y_predict))
print()
print("F1 Score:",f1_score(y_test,y_predict))
print()
print("Roc Auc Score:",roc_auc_score(y_test,y_predict))
5 Decision Tree Model:¶
Note - In Decision Trees, we need not remove highly correlated variables as nodes are divided into sub nodes using one independent variable only, hence even if two or more variables are highly correlated, the variable producing the highest information gain will be used for the analysis
In [71]:
# splitting data into training and test set for independent attributes
from sklearn.model_selection import train_test_split
features = [col for col in df_tree.columns if col != 'Target']
X_train, X_test, y_train, y_test = train_test_split(df_tree[features], df_tree['Target'], test_size=.3, random_state=22)
X_train.shape, X_test.shape
Out[71]:
Building a Decision Tree¶
In [72]:
# Invoke the decision tree classifier function.
# create gini and entropy methods of finding the split columns
model_entropy = DecisionTreeClassifier(criterion='entropy', random_state=42)
model_gini = DecisionTreeClassifier(criterion='gini', random_state=42)
In [73]:
model_entropy.fit(X_train, y_train)
model_gini.fit(X_train, y_train)
Out[73]:
In [74]:
treeObj = model_entropy.tree_
print ('Total Nodes:',treeObj.node_count)
print('Depth:',model_entropy.get_depth())
print('Total Leaves:',model_entropy.get_n_leaves())
In [75]:
treeObj = model_gini.tree_
print ('Total Nodes:',treeObj.node_count)
print('Depth:',model_gini.get_depth())
print('Total Leaves:',model_gini.get_n_leaves())
In [76]:
print("Entropy Model Train: %.2f" % model_entropy.score(X_train, y_train)) # performance on train data
print("Entropy Model Test: %.2f" % model_entropy.score(X_test, y_test)) # performance on test data
print("Gini Model Train: %.2f" % model_gini.score(X_train, y_train)) # performance on train data
print("Gini Model Test: %.2f" % model_gini.score(X_test, y_test)) # performance on test data
Findings¶
There is a high degree of overfitting in the model due to which the test accuracy drops drastically. This shows why decision trees are prone to overfitting.
Regularize/prune the decision tree by limiting the max. depth of trees and print the accuracy.
There is no statistical significance on the values of depth and minimum samples that is considered below. The idea is to prune a tree and check how pruning impacts accuarcy.
In [77]:
clf_pruned = DecisionTreeClassifier(criterion = "gini", max_depth=5, min_samples_leaf=5, random_state=42)
clf_pruned.fit(X_train, y_train)
Out[77]:
In [78]:
print("Train: %.2f" % clf_pruned.score(X_train, y_train)) # performance on train data
print("Test: %.2f" % clf_pruned.score(X_test, y_test)) # performance on test data
Visualizing the tree¶
In [79]:
import io
from io import StringIO
from sklearn.tree import export_graphviz
from IPython.display import Image
import pydotplus
import graphviz
feature_cols = X_train.columns
In [80]:
dot_data = StringIO()
export_graphviz(clf_pruned, out_file=dot_data,
filled=True, rounded=True,
special_characters=True,feature_names = feature_cols,class_names=['5','6','8'])
In [81]:
from pydotplus import graph_from_dot_data
#(graph, ) = graph_from_dot_data(dot_data.getvalue())
graph = graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
Out[81]:
In [82]:
preds_train = clf_pruned.predict(X_train)
preds_test = clf_pruned.predict(X_test)
acc_DT = accuracy_score(y_test, preds_test)
Decision tree performance¶
In [83]:
# Confusion matrix
pd.crosstab(y_test, preds_test, rownames=['Actual'], colnames=['Predicted'])
Out[83]:
In [84]:
# Model validation Score
print("Trainig accuracy",clf_pruned.score(X_train,y_train))
print()
print("Testing accuracy",clf_pruned.score(X_test, y_test))
print()
print("----------------------Micro-----------------------------------------")
print("Recall:",recall_score(y_test,preds_test, average="micro"))
print()
print("Precision:",precision_score(y_test,preds_test, average="micro"))
print()
print("----------------------Macro-----------------------------------------")
print("Recall:",recall_score(y_test,preds_test, average="macro"))
print()
print("Precision:",precision_score(y_test,preds_test, average="macro"))
print()
In [85]:
# Visualize model performance with yellowbrick library
viz = ClassificationReport(DecisionTreeClassifier(criterion = "gini", max_depth=3, min_samples_leaf=4))
viz.fit(X_train, y_train)
viz.score(X_test, y_test)
viz.show()
roc = ROCAUC(DecisionTreeClassifier(criterion = "gini", max_depth=3, min_samples_leaf=4))
roc.fit(X_train, y_train)
roc.score(X_test, y_test)
roc.show()
Out[85]:
The regualarized tree shows similar accuracy score for both training and test data, however accuracy doesn't improve much¶
In [86]:
## calculate feature importance
feat_importance = clf_pruned.tree_.compute_feature_importances(normalize=False)
feat_imp_dict = dict(zip(features, clf_pruned.feature_importances_))
feat_imp = pd.DataFrame.from_dict(feat_imp_dict, orient='index')
feat_imp.sort_values(by=0, ascending=False).head()
Out[86]:
From the feature importance dataframe we can infer that duration, housing, contact_unknown, month, andage are the variables that impact Target¶
In [87]:
#Store the accuracy results for each model in a dataframe for final comparison
resultsDf = pd.DataFrame({'Method':['Decision Tree - Gini - Pruned'], 'accuracy': acc_DT})
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[87]:
6 Ensemble Model:¶
Apply the Random forest model and print the accuracy of Random forest Model¶
In [88]:
from sklearn.ensemble import RandomForestClassifier
rfcl = RandomForestClassifier(n_estimators = 10, max_samples=0.8, random_state=42)
rfcl = rfcl.fit(X_train, y_train)
rfcl
Out[88]:
In [89]:
pred_RF = rfcl.predict(X_test)
acc_RF = accuracy_score(y_test, pred_RF)
In [90]:
tempResultsDf = pd.DataFrame({'Method':['Random Forest'], 'accuracy': [acc_RF]})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[90]:
In [91]:
## Calculating feature importance
feat_importance = rfcl.feature_importances_
feat_imp_dict = dict(zip(features, rfcl.feature_importances_))
feat_imp = pd.DataFrame.from_dict(feat_imp_dict, orient='index')
feat_imp.sort_values(by=0, ascending=False)
Out[91]:
In [92]:
rfcl.estimators_[5]
Out[92]:
In [93]:
dot_data = StringIO()
export_graphviz(rfcl.estimators_[5], out_file=dot_data,
filled=True, rounded=True,
special_characters=True,feature_names = features,class_names=['5','6','8'])
from pydotplus import graph_from_dot_data
#(graph, ) = graph_from_dot_data(dot_data.getvalue())
graph = graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
Out[93]:
Apply the Bagging model and print the accuracy of Random forest Model¶
In [94]:
from sklearn.ensemble import BaggingClassifier
bgcl = BaggingClassifier(n_estimators=70, max_samples= .8, bootstrap=True, random_state=22)
#------bootstrap=True signifies that entire sample is not used to develop a tree
bgcl = bgcl.fit(X_train, y_train)
In [95]:
pred_BG = bgcl.predict(X_test)
acc_BG = accuracy_score(y_test, pred_BG)
In [96]:
tempResultsDf = pd.DataFrame({'Method':['Bagging'], 'accuracy': [acc_BG]})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[96]:
Apply the Boosting model and print the accuracy of Random forest Model¶
Ada-Boost¶
In [97]:
from sklearn.ensemble import AdaBoostClassifier
abcl = AdaBoostClassifier(n_estimators = 100, learning_rate=0.1, random_state=22)
abcl = abcl.fit(X_train, y_train)
# n_estimator - The maximum number of estimators at which boosting is terminated. In case of perfect fit, the learning procedure is stopped early.
In [98]:
pred_AB =abcl.predict(X_test)
acc_AB = accuracy_score(y_test, pred_AB)
In [99]:
tempResultsDf = pd.DataFrame({'Method':['Adaboost'], 'accuracy': [acc_AB]})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[99]:
Gradient-Boost¶
In [100]:
from sklearn.ensemble import GradientBoostingClassifier
gbcl = GradientBoostingClassifier(n_estimators = 50, learning_rate = 0.1, random_state=22)
gbcl = gbcl.fit(X_train, y_train)
In [101]:
pred_GB = gbcl.predict(X_test)
acc_GB = accuracy_score(y_test, pred_GB)
In [102]:
tempResultsDf = pd.DataFrame({'Method':['Gradient Boost'], 'accuracy': [acc_GB]})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[102]:
In [103]:
x=model_entropy.predict_proba(X_train)
x[1,1]
Out[103]:
In [104]:
x=clf_pruned.predict_proba(X_test)
x[250,1]
Out[104]:
Grid Search¶
In [105]:
from sklearn.model_selection import GridSearchCV
params = {'criterion':['gini','entropy'],'max_samples':[0.7,0.75,0.6,0.8],'bootstrap':[True],'n_estimators':list(range(10,120,20))} #------------------Setting parameter values for Decision Tree
grid_search_model = GridSearchCV(RandomForestClassifier(random_state=42), params, verbose=1, cv=3) #---Cross validation is 3
grid_search_model.fit(X_train, y_train)
Out[105]:
In [106]:
#-------------------------Getting the estimator the returns the best fit model based on accuracy score--------------------------
grid_search_model.best_estimator_
Out[106]:
In [107]:
grid_search_model.best_params_
Out[107]:
In [108]:
from sklearn.ensemble import RandomForestClassifier
rfcl = RandomForestClassifier(n_estimators = 70, random_state=42, criterion = 'entropy', max_samples=0.8)
rfcl = rfcl.fit(X_train, y_train)
display(rfcl)
In [109]:
pred_RF = rfcl.predict(X_test)
acc_RF = accuracy_score(y_test, pred_RF)
In [110]:
tempResultsDf = pd.DataFrame({'Method':['Random Forest-Grid Search'], 'accuracy': [acc_RF]})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Method', 'accuracy']]
resultsDf
Out[110]:
In [111]:
# screenshot of Final Logistic Regression model performance for reference / comparison against other models
from IPython.display import Image
Image("lr_finalModelScore.jpg")
Out[111]:
7 Model Comparisons:¶
Highest Accuracy = "Random Forest - Grid Search" Lowest Accuracy = Logistic Regression followed by Adaboost Delta between Highest and Lowest Accuracy = .12
8 Conclusions:¶
Run - Decision Tree Gini / Pruned Model for predeictions for its simplicity and interpretability compared to the complexity of all ensemble methods for marginal gains in performance.
In [ ]: